ABSTRACT
The COVID-19 pandemic has made a dramatic impact on human life, medical systems, and financial resources. Due to the disease's pervasive nature, many different and interdisciplinary fields of research pivoted to study the disease. For example, deep learning (DL) techniques were employed early to assess patient diagnosis and prognosis from chest radiographs (CXRs) and computed tomography (CT) scans. While the use of artificial intelligence (AI) in the medical sector has displayed promising results, DL may suffer from lack of reproducibility and generalizability. In this study, the robustness of a pre-trained DL model utilizing the DenseNet-121 architecture was evaluated by using a larger collection of CXRs from the same institution that provided the original model with its test and training datasets. The current test set contained a larger span of dates, incorporated different strains of the virus, and included different immunization statuses. Considering differences in these factors, model performance between the original and current test sets was evaluated using area under the receiver operating characteristic curve (ROC AUC) [95% CI]. Statistical comparisons were performed using the Delong, Kolmogorov-Smirnov, and Wilcoxon rank-sum tests. Uniform manifold approximation and projection (UMAP) was used to help visualize whether underlying causes were responsible for differences in performance between test sets. In the task of classifying between COVID-positive and COVID-negative patients, the DL model achieved an AUC of 0.67 [0.65, 0.70], compared with the original performance of 0.76 [0.73, 0.79]. The results of this study suggest that underlying biases or overfitting may hinder performance when generalizing the model. © 2023 SPIE.